Brain interaction during cooperation: Evaluating local properties of multiple-brain network

Subjects’ interaction is the core of most human activities. This is the reason why a lack of coordination is often the cause of missing goals, more than individual failure. While there are different subjective and objective measures to assess the level of mental effort required by subjects while facing a situation that is getting harder, that is, mental workload, to define an objective measure based on how and if team members are interacting is not so straightforward. In this study, behavioral, subjective and synchronized electroencephalographic data were collected from couples involved in a cooperative task to describe the relationship between task difficulty and team coordination, in the sense of interaction aimed at cooperatively performing the assignment. Multiple-brain connectivity analysis provided information about the whole interacting system. The results showed that averaged local properties of a brain network were affected by task difficulty. In particular, strength changed significantly with task difficulty and clustering coefficients strongly correlated with the workload itself. In particular, a higher workload corresponded to lower clustering values over the central and parietal brain areas. Such results has been interpreted as less efficient organization of the network when the subjects’ activities, due to high workload tendencies, were less coordinated

Development of a System for the Training Assessment and Mental Workload Evaluation

Several studies have demonstrated that the main cause of accidents are due to Human Factor (HF) failures. Humans are the least and last controllable factor in the activity workflows, and the availability of tools able to provide objective information about the user’s cognitive state should be very helpful in maintain proper levels of safety. To overcome these issues, the objectives of the PhD covered three topics. The first phase was focused on the study of machine-learning techniques to evaluate the user’s mental workload during the execution of a task. In particular, the methodology was developed to address two important limitations: i) over-time reliability (no recalibration of the algorithm); ii) automatic brain features selection to avoid both the underfitting and overfitting problems. The second phase was dedicated to the study of the training assessment. In fact, the standard training evaluation methods do not provide any objective information about the amount of brain activation\resources required by the user, neither during the execution of the task, nor across the training sessions. Therefore, the aim of this phase was to define a neurophysiological methodology able to address such limitation. The third phase of the PhD consisted in overcoming the lack of neurophysiological studies regarding the evaluation of the cognitive control behaviour under which the user performs a task. The model introduced by Rasmussen was selected to seek neurometrics to characterize the skill, rule and knowledge behaviours by means of the user’s brain activity. The experiments were initially ran in controlled environments, whilst the final tests were carried out in realistic environments. The results demonstrated the validity of the developed algorithm and methodologies (2 patents pending) in solving the issues quoted initially. In addition, such results brought to the submission of a H2020-SMEINST project, for the realization of a device based on such results

A new perspective for the training assessment: Machine learning-based neurometric for augmented user's evaluation

Inappropriate training assessment might have either high social costs and economic impacts, especially in high risks categories, such as Pilots, Air Traffic Controllers, or Surgeons. One of the current limitations of the standard training assessment procedures is the lack of information about the amount of cognitive resources requested by the user for the correct execution of the proposed task. In fact, even if the task is accomplished achieving the maximum performance, by the standard training assessment methods, it would not be possible to gather and evaluate information about cognitive resources available for dealing with unexpected events or emergency conditions. Therefore, a metric based on the brain activity (neurometric) able to provide the Instructor such a kind of information should be very important. As a first step in this direction, the Electroencephalogram (EEG) and the performance of 10 participants were collected along a training period of 3 weeks, while learning the execution of a new task. Specific indexes have been estimated from the behavioral and EEG signal to objectively assess the users' training progress. Furthermore, we proposed a neurometric based on a machine learning algorithm to quantify the user's training level within each session by considering the level of task execution, and both the behavioral and cognitive stabilities between consecutive sessions. The results demonstrated that the proposed methodology and neurometric could quantify and track the users' progresses, and provide the Instructor information for a more objective evaluation and better tailoring of training programs. © 2017 Borghini, Aricò, Di Flumeri, Sciaraffa, Colosimo, Herrero, Bezerianos, Thakor and Babiloni

Passive BCI in operational environments: insights, recent advances and future trends

this mini-review aims to highlight recent important aspects to consider and evaluate when passive Brain-Computer Interface (pBCI) systems would be developed and used in operational environments, and remarks future directions of their applications

EEG-based mental workload neurometric to evaluate the impact of different traffic and road conditions in real driving settings

Car driving is considered a very complex activity, consisting of different concomitant tasks and subtasks, thus it is crucial to understand the impact of different factors, such as road complexity, traffic, dashboard devices, and external events on the driver’s behavior and performance. For this reason, in particular situations the cognitive demand experienced by the driver could be very high, inducing an excessive experienced mental workload and consequently an increasing of error commission probability. In this regard, it has been demonstrated that human error is the main cause of the 57% of road accidents and a contributing factor in most of them. In this study, 20 young subjects have been involved in a real driving experiment, performed under different traffic conditions (rush hour and not) and along different road types (main and secondary streets). Moreover, during the driving tasks different specific events, in particular a pedestrian crossing the road and a car entering the traffic flow just ahead of the experimental subject, have been acted. A Workload Index based on the Electroencephalographic (EEG), i.e., brain activity, of the drivers has been employed to investigate the impact of the different factors on the driver’s workload. Eye-Tracking (ET) technology and subjective measures have also been employed in order to have a comprehensive overview of the driver’s perceived workload and to investigate the different insights obtainable from the employed methodologies. The employment of such EEG-based Workload index confirmed the significant impact of both traffic and road types on the drivers’ behavior (increasing their workload), with the advantage of being under real settings. Also, it allowed to highlight the increased workload related to external events while driving, in particular with a significant effect during those situations when the traffic was low. Finally, the comparison between methodologies revealed the higher sensitivity of neurophysiological measures with respect to ET and subjective ones. In conclusion, such an EEG-based Workload index would allow to assess objectively the mental workload experienced by the driver, standing out as a powerful tool for research aimed to investigate drivers’ behavior and providing additional and complementary insights with respect to traditional methodologies employed within road safety research

EEG-based cognitive control behaviour assessment: an ecological study with professional air traffic controllers

Several models defining different types of cognitive human behaviour are available. For this work, we have selected the Skill, Rule and Knowledge (SRK) model proposed by Rasmussen in 1983. This model is currently broadly used in safety critical domains, such as the aviation. Nowadays, there are no tools able to assess at which level of cognitive control the operator is dealing with the considered task, that is if he/she is performing the task as an automated routine (skill level), as procedures-based activity (rule level), or as a problem-solving process (knowledge level). Several studies tried to model the SRK behaviours from a Human Factor perspective. Despite such studies, there are no evidences in which such behaviours have been evaluated from a neurophysiological point of view, for example, by considering brain activity variations across the different SRK levels. Therefore, the proposed study aimed to investigate the use of neurophysiological signals to assess the cognitive control behaviours accordingly to the SRK taxonomy. The results of the study, performed on 37 professional Air Traffic Controllers, demonstrated that specific brain features could characterize and discriminate the different SRK levels, therefore enabling an objective assessment of the degree of cognitive control behaviours in realistic setting

Multivariate model for cooperation: bridging Social Physiological Compliance and Hyperscanning

The neurophysiological analysis of cooperation has evolved over the past 20 years, moving towards the research of common patterns in neurophysiological signals of people interacting. Social Physiological Compliance (SPC) and Hyperscanning represent two frameworks for the joint analysis of autonomic and brain signals respectively. Each of the two approaches allows to know about a single layer of cooperation according to the nature of these signals: SPC provides information mainly related to emotions, and Hyperscanning that related to cognitive aspects. In this work, after the analysis of the state of the art of SPC and Hyperscanning, we explored the possibility to unify the two approaches creating a complete neurophysiological model for cooperation considering both affective and cognitive mechanisms. We synchronously recorded electrodermal activity, cardiac and brain signals of 14 cooperative dyads. Time series from these signals were extracted and Multivariate Granger Causality was computed. The results showed that only when subjects in a dyad cooperate there is a statistically significant causality between the multivariate variables representing each subject. Moreover, the entity of this statistical relationship correlates with the dyad's performance. Finally, given the novelty of this approach and its exploratory nature, we provided its strengths and limitations

The application of human factors in wake vortex encounter flight simulations for the reduction of flight upset risk and startle response

The current top safety risk concern for commercial air travel in Europe is known as “Flight Upset”. This term, also known as “Loss of Control in Flight”, entails the flight crew suddenly finding themselves in an unexpected, complex, and even confusing situation that if not resolved quickly can lead to a major accident. Accidents such as AF447 and the two B737 Max accidents fall into this category. An undesirable aspect of such events is known as the “startle response”, wherein one or both flight crew, finding themselves in dire and dangerous conditions, may experience ‘startle’, which temporarily affects their cognitive functioning. This may only last half a minute, but its effect can have a severe impact on the survivability of such events. A Horizon 2020 research project called SAFEMODE, which aims to integrate Human Factors techniques into a unified framework for designers in aviation and maritime domains, is exploring the use of state-of-the-art flight simulation facilities to measure pilot performance in severe wake turbulence events, which can induce the startle effect. This is part of a broader use case within SAFEMODE to validate the design of a new Wake Vortex Air Traffic Alert for the Cruise phase of flight. A tactical short-term alert to the Flight Crew, ahead of the wake encounter, is seen as beneficial to reduce the startle effect and support the appropriate management of these conflicts. The envisaged risk-alerting logic relies on a ground-based predictor, connected to the Air Traffic Control system, displaying an alert to the En-route Air Traffic Controllers, who can then provide a cautionary advisory to the Flight Crew so they can take appropriate actions.The cockpit flight simulations involve type-rated flight crews in realistic and representative cruise flight conditions, using a Type VI Boeing 737-800 full flight motion-based simulator (also used for Upset Prevention and Recovery training programs). During the simulation runs, pilots are exposed to simulated wake vortex encounters, corresponding to a strong wake-induced upset (between 30 and 40 degrees of bank), with or without prior ATC wake caution, and varying the initial direction of roll between left and right to limit the simulation training effect.Human Factors measurements include workload, situation awareness, trust, acceptability-based user feedback, as well as psychophysiological measures such as eye-tracking and Electro-Dermal Activity (EDA). In particular, eye-tracking is expected to support the refined determination of the sequence of actions before and after detection, and the reaction of flight crews to the en-route ATC Wake alert.A cockpit flight simulation, via combining the analyses of psychophysiological measures, flight parameters, expert observations and subjective pilot feedback, enables evaluation of Flight Crews performance in preparing for, managing or avoiding wake encounter upsets with the new ATC wake alerts, showing the net safety benefits. Early results indicate that the simulations can indeed induce startle effect, and that repeated exposure enables flight crew to overcome it and manage the situation in a more measured and controlled fashion

A New Perspective for the Training Assessment: Machine Learning-Based Neurometric for Augmented User's Evaluation

Inappropriate training assessment might have either high social costs and economic impacts, especially in high risks categories, such as Pilots, Air Traffic Controllers, or Surgeons. One of the current limitations of the standard training assessment procedures is the lack of information about the amount of cognitive resources requested by the user for the correct execution of the proposed task. In fact, even if the task is accomplished achieving the maximum performance, by the standard training assessment methods, it would not be possible to gather and evaluate information about cognitive resources available for dealing with unexpected events or emergency conditions. Therefore, a metric based on the brain activity (neurometric) able to provide the Instructor such a kind of information should be very important. As a first step in this direction, the Electroencephalogram (EEG) and the performance of 10 participants were collected along a training period of 3 weeks, while learning the execution of a new task. Specific indexes have been estimated from the behavioral and EEG signal to objectively assess the users' training progress. Furthermore, we proposed a neurometric based on a machine learning algorithm to quantify the user's training level within each session by considering the level of task execution, and both the behavioral and cognitive stabilities between consecutive sessions. The results demonstrated that the proposed methodology and neurometric could quantify and track the users' progresses, and provide the Instructor information for a more objective evaluation and better tailoring of training programs

Audio Focus: Interactive spatial sound coupled with haptics to improve sound source location in poor visibility

International audienceIn an effort to simplify human resource management and reduce costs, control towers are now more and more designed to not be implanted directly on the airport but remotely. This concept, known as Remote Control Tower, offers a “digital” working context because the view on the runways is broadcast remotely via cameras, which are located on the physical airport. This offers researchers and engineers the possibility to develop novel interaction techniques. But this technology relies on the sense of sight, which is largely used to give the operator information and interaction, and which is now becoming overloaded. In this paper, we focus on the design and the testing of new interaction forms that rely on the human senses of hearing and touch. More precisely, our study aims at quantifying the contribution of a multimodal interaction technique based on spatial sound and vibrotactile feedback to improve aircraft location. Applied to Remote Tower environment, the final purpose is to enhance Air Traffic Controller's perception and increase safety. Three different interaction modalities have been compared by involving 22 Air Traffic Controllers in a simulated environment. The experimental task consisted in locating aircraft in different airspace positions by using the senses of hearing and touch through two visibility conditions. In the first modality (spatial sound only), the sound sources (e.g. aircraft) had the same amplification factor. In the second modality (called Audio Focus), the amplification factor of the sound sources located along the participant's head sagittal axis was increased, while the intensity of the sound sources located outside this axis was decreased. In the last modality, Audio Focus was coupled with vibrotactile feedback to indicate in addition the vertical positions of aircraft. Behavioral (i.e. accuracy and response times measurements) and subjective (i.e. questionnaires) results showed significantly higher performance in poor visibility when using Audio Focus interaction. In particular, interactive spatial sound gave the participants notably higher accuracy in degraded visibility compared to spatial sound only. This result was even better when coupled with vibrotactile feedback. Meanwhile, response times were significantly longer when using Audio Focus modality (coupled with vibrotactile feedback or not), while remaining acceptably short. This study can be seen as the initial step in the development of a novel interaction technique that uses sound as a means of location when the sense of sight alone is not enough